DOCUMENT RESUME 



ED 217 053 

AUTHOR 
TITLE 



INSTITUTION 
SPONS' XGENCT 
PUB DATE 
CONTRACT 
NOTE 



/ 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



' TM 820 240 

Lawrei(ice, John E.-ST7~Cook, T;homas J? 

Incoi;]porating vin format ion Needs of Key Constituencies 

in Program. Evaluation Design: An Empirical 

Approach. i ' L 

Research Triangle Inst., Durham, N.C. 
Department of Agriculture, Washington, D.C. 
1 Oct 81 
53-3198-9-87 ^ 

32p.; Paper presented at the. Joint Meeting of 
• Evaluation Network and Evaluation. Research Society 
(Austin, TX/ October. 1-3, 1981); Marginal legibility 
due to broken type. 

MF01/PC02 Plus Postage. 

Decision Making; *Evaluation Methods; * Information 
Needs;. Information Utilization; *Program Evaluation; 
♦Research Design; *Research Methodology; *Surveys 
Evaluability Assessment; *Stakeholder Survey 



'^ABSTRACT 

As the procedures of evaluation research. have 
developed during refcent years, it.has become clear that the results 
of evaluation studies have failed to provide policy decision makers 
with useful information. An approach towards designing and 
implementing more useful evaluations— the stakeholder survey — is 
discussed. Stakeholders are individuals, or groups, who have a direct 
interest in the program being evaluated. The central premise of this 
paper is that the information needs of stakeholders should be 
incorporated directly into the formulatjlon of evaluation objectives, 
and the design of the evaluatioa to achieve those objectives. A 
/rationale for, the stakeholder survey is presented, and its three 
' primary functions (identification, contact and information synth^si-s; 
ai;e described, as are application of the approach in a multi-year 
eV^aluation ot a' national supplemental food program. Also considered 
are key issues involvad in applying the approach. The evidence from 
this study suggests that such surveys are not only feasible, but also 
of considerable value to the evaluation team. (Author/AL) 
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- • The failure of program evaluaLions- to provide- |i{O.Ucy- dc-ci si on 
makers with useful ijiformatibn is a familiar complaint. This paper - 
discusses an approaclv towards designing and implementing more useful 
evaluations--the stakeholder survey. Stakeholders are individuals, or 
g'roups, who have a direct interest in the program being evaluated. The 
central premise of the paper is that the information needs of stake- 
holders should be incorporated directly into the formulation of evalua- 
tion objectives, and the design of the evaluation to achieve those 
objectives. The paper presents a rationale for the stakeholder survey, 
describes its major components, and reports oh the application of the 
approach in a large^program evaluation. 'Key issues involved in applying 
•the approach are also considered. The e\)idence from this study suggests 
that such surveys are not only feasible, /but also of considerable value 
to the evaluation team. 
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. DESIGKINU USEhUL EVALUAIIOHS; HIE SIAKEHOLDEK SURVEY 
_ _ " ' 1. imiWCTlON 

Social services programs involve both individuals and organizations 

^ ' ' 

having interests or "stakes" of various kinds in program operations and 
outcomes.. Collectively, this set of constituencies is sometimes ref- 
erred to as the "s:J;akeholders" . for the program, and includes those who 

pay for, carry ^ut, or receive program treatments. It follows that 

<^ ' ' > ^ 

Jthgse_iame_^^ can be characterized as sta^e- 




As the procedures of evaluation research have developed during 
recent years, it has- become clear that the results of evaluation studies 
may be litl^le used because they failed to provide answers to questions 
trouBling key stakeholders. Hence, it is now considered desirable, and 
possibly imperative, to asses.s in some depth what various stakeholders \ 
expect and n^ed from a g.iven program evaluation. Accordingly, evalua- 
tion researchers have emph.asized that, as part of predesign efforts' 
(sjjch as literature reviews, data-base reviews, ^ivaluability assessments 
"etc.), research resources should be directed towards identification of 
relevant program constituencies, and various mechanisms for receiving 
and using stakeholder input during the formula^ ve stages of an evalua- 
tion. ' ' \ 

This paper presents a rationale for such formal assessment of 

stakeholder information needs, examines the theoretical background and 

I 

available methodologies, and discusses the implementation of one approach 
in a survey recently conducted as part of a major federal program evalua- 
tion,* The proce^i is briefly described, methodological^ issues presented 

■\ \ , ■\ 

~ ^Research for ttiis paper was conducted under USDA Contract No. 
53-3198-9-87. Appreci\ation is expressed for review comments provided by 
Dr. W. Burleigh Seaver of the Office of Policy Planning and Evaluation, 
Food and Nutrition Service USDA. 



aiTO rinatly, the iiiiplications .ue d'rscussed ofnchis research tool Tor 
evaluation research, ^ 
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11; BACKGROUND ANU RAIIONAU i-Ull SIAKLHOLUhR INILRVILWS \ 
"in LVALUAIION RLSLARCII 



The last 20 years have seen rapid growth in evaluation research, 
' particularly in government ^^sponsored studies on public social programs 
(Rossi, 1979). One consequence of Ihis growth is wider interdiscipli- 
nary interest in the conceptually methodological, and operational prob- 
lemS of program evaluation, and the corresponding e^daptation or gene- 
ration of methods or approaches from a variety of social science per- 
spectives (Davis and Salasin, 1979; lanni and Orr, 1979). Moreover, 
considerable emphasis is currently placed on the predesign phase^ of 
program evaluation (Ruttman, 1977). The predesign phase is so called 
not only because it precedes final ization of the actual design to be 
used in' the study, but because it essentially informs the selection or 
refinement of research approaches originally proposed by the evaluators. 
Predesign activities are directed towards constructing the evalua- 



tion so as to maximize the probability of scientifically credible re- 
search results of .practical value to decision makers. In woTrking' to 
s^tructure-the research so itntray be mare Useful fo~rl)"ro"gfam ■pbTicy'"arra 



management decisions, evaluators should recognize the complex and fre-. 
quently changing program environment within which they operate. And 
yet, the absence of attention to the dynamic ^political and operational 
contexts '|)oth of program operations, and of evaluations themselves as 
separate enterprises, has repeatedly been noted (Weiss, 1972; Levine', 
1977; Grant, 1978; Rossi, Freeman, and Wright, 1979; Who ley, 1979). 
For example, Scanlon et al. (1977) characterized two major failings of 



evaluation as follows: 



\ 



\ 
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statisticians wony^'about two types of "errors,. . • J \ 

Type 1 e rror Is .rcjeAi^U-^i--%+>f^^:^^^ ^- — 

should be accepted;" Ivpe II error is accepting a 
tiypothesis When\it should be rejected. Evalluators 
commonly .make two. types of erroi^s in doing evalua- 
\ ti'ons: Ty|ye" Ill^error is measuring something that 

*^ does not exist; and Type IV error is measuring some- 

thing 'that is of j(\o interest to management and 
policy, makers. 

Evaluation researchers would therefore seem justified in directing part 
of their resources, towards avoiding these kinds of mistakes. As Carol 
Weiss notes: 

\ "Only when the evaluator has insight fnto the in- 
terests and motivations of the other actors in the 
system, understands the roles that [s/he]..is con- 
sciously or inadvertently playing-, realizing the 
obstacles and opportunites tha^t impinge upon the^ 
evaluative effort, and. the limitations and possi- 
bilities for putting the results 6,f the evaluation 
to work—only with sensitivity to the politics of 
evaluation reseairch can the evaluator be as creative 
. and strategically useful as [s/he] should be" (1975, 
^ • p. 14). 

More recently, in addressing t-he-need~f-or--eva-liiatit)ir-s-tandardS7--;the 

case was again made for ^understanding the complex setting into which 

each evaluation should be individually framed (Baron, |981). 

Frequently, evaluators have failed to acknowledge 
the importance of the social and political context \ 
within whicri the program operates and the evaluation 
must be conducted. The failure to recognize and act 
^ upon these dimensions have too often both undermined 
the success of the evaluation study and jeopardized 
its potential utility...-* (p. 165). 

The "Value of including decision makers' interests in evaluation 

design considerations has also been emphasized by those agencies sppn- 

soring and .monitoring evaluations in the public sector (Berlin, 1977; 

General AccountinglOffice, 1976). The^user oriented approach in evalu- 

ation is described by Patton (1978) as discriminating between evalua- 



tions which are directly relevant, to subsequent decision making and 
thos^ which are hot. , , 
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le JocUs in Llie iiLi I ii.aLion- 1 ociised. approach -Lo 
Ml4iairio»-t-es^?nr€lHVt ) ii 1 dun li f i c<i I i un ami or- 



. ganization of relevant decisjioa makers and infor- 

\^^l>^v maUon users, • •Whore siidv a person or group was 

present, evaluations were used; where the personal 

^ /. factor was absent; therelwas a correspondingly 

— marked absence of evaluation impact^^.The reason for 

identify';ng and organizing relevant decision makers 

,and informe^tion users is to be sure Ithat the people ^ 

who are goijig to be the primary users of evaluation 

findings are the same people who decide what the 

focus of the evaluation will be. This means that 

the evaluation should focus on their information ' t 

. needs— not on their speculations about what someone 

else wants to know (p. 64). 
♦ • ■» . 

Furthermore, the tapping of multiple perspectives and data sources 
in program Evaluation to capture the most .comprehensive view of the 
social intervention being studied is the basis of the strategy of "data 
triangulation">in evaluation research (Denzin, 1970;^ Reichardt and Cook, 
1979). Finally, attention to the context of program evaluation before 
finalizing evaluation design strategies reflects a recent trend in 
,eva1 uatloD~Jceseanch---Xowaf-ds linJcing--fxr^cess--analysis-4v44;h- 1mpacU-an= — 



\ 

alysis, and towards emphasizing the "narrative history" sections of 
evaluation research reports as essential to the planning ano execution 



of research studies, rather than as- cosmetic afterthoughts (Campbell, 
1979). As a result, standards have been, recommended (Standards, 1980) 
reflectina ^ general consensus concerning the need for program evalua- 
tions tvc responsive to client interests, with specific recommenda- 
tions for audience identification at the outset. , 

However, while there seems to be a prescription in the literature 
for achieving at least a working consensus among major parties to an 
evaluation on the scope,, political environment, expectations and ex- 
, pected outcomes of the research » there is little empirical gui (dance on 
how' to accomplish the task. For example, al'though Cook and Campbell 
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"tll^/STT^^T'W^T^Hly definl' as a "crucial evaluation issue..." the 



incorporation "into the evaluation of concerns of various constituencies 
with an interjsst in the eventual findings,"' no discussion of methods or 
supporting data are included.^ 

"Evaluability assessment!' is one kind of predesign activity that is 
intended to assist in defining, for example, original program purpose, . 
current program purpose, and the perceptions of and reasons for dif- . 
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ferences, if any, between them; the purposes of an evaluation of the 
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program, potential users of evaluative information; the actual expecta- 
tions of these identified users for the outcomes^ot the evaluation, and 
' where appropriate, specific information needs of such users. Evalua- 

^ bility assessment is also concerned with many other tasks, such as the 

< 

construction of "rhetorical" vs. "actual" program models, assessment of 
program data availability, and other dimensions ^of evaluative feasibil- 
it y (WhoteyT-197:7, 1 97 9 ; R utman-r^-98e^^ — The-iJart-of^-evaluahjOily^as- 



sessment that is of particular interei^t here, however, is that in which 
-are-determined the major dimensions of program impact of interest to 
policy makers, the existing and anticipated policy Context both of the 
program during the span of the evaluation and of the evaluation itself, 

.and finally, the specific information needs of policy makers that are 

1 

expected to be addressed by the. evaluation study. Developing these 
kinds of predesign data requires that relevant groups of policy decision 
makers.be firstly identified ,- and secondly contacted appropriately, and 
that thirdly the information gathered be synthesized adequately and in 
time to be incorporated "into the eventual study design. Many purposes 
are served by this approach. 
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HollisLer el al* (19/9) note thai pfograjn staff are almost always 



convinced "of the effectiveness, of .the program they are running, such 
convictioa being necessary for continued high motivation. Pre-evalua- 
tion discussion between evaluators and program administrators or advo- 
cates (or opponents) can aTsist in surfacing[^ these biases and hypothe- 

^ical reasons for, them, reduce the naivete of the researchers, and 

/ 

alleviate the danger of the "ivory tower", posture toward program evalua- 

if . ^ 

tion sometimes attributed to outside ^valuators. Conversely, ^the ex- 
planation by evaluators to key policy makers of the nature, duration, 

and'type of evaluation prior to the -potential programmatic disruption of 

i • • t / 

evaluative data gathering ef forts , can inform evaluators regarding opera- 

tional aspects of the desjgn (e.g(, initial approaches to state and 
local administrators*, on-site data gathering and quality control proce- 
dures), and reduce the threat of evaluation and suspicion of evaluators 
encountered among program administrators and participants (Knapp, 1979a; 
lanni and Orr, 1979). 

The three primary. functions of the stakeholder survey, (identifica- 
tion, contact and'information synthesis) are discussed separately in the 
following sections. 

A. Identification of Stakeholder Set 

Selection of the specific method for identification of key constit- 
juencies depends updn several factors, including the resources for pre- 
design activities, the receptivity" of the sponsoring.,agency to multiple 
uredesign points of view, and the political salience of the program. 

J) 

Currently, however, there are only scattered references to methodology 
for surveying stakeholders, and no recognized or systematic approach 
exists. At best, guiding concepts can be borrowed from evaluability 



asscs5inonis; "JpeirsysLem" Lhuonos u( oryanizaLions as inleracling 
continualTy with relevant environments, and more particularly from 
management strategies for, revealing assumptions in organizational prob- 
.lem solving. Wholey et al,. (1975, -1977, 1979) have developed a number 
of techniques for soliciting information on stated ffrogram purposes from 
relevant program administrators and ^pthers in the immediate program^ 
policy environment. The major thrust of thei'r approach is towards the 
construction of "rhetorical model?" of program operation that exp'ose the 

assumed^ linkages between program treatment and program effects. By 

- > ' ' ^ *^ 

eliminating unrealistic or untestable assumptions, an evaluable program 

% 

model and testable hypotheses result. Empirical evidence of this ap- 
■proach is concentrated mostly however on program management as the user, 
and there is less guidance as to methods for accessing relevant 'others* 
or affected parties in the program environment.. 

Krause and Howard (1976) introduce the concept of "multiple other 
parties" in the evaluation context, and the need-to^take the interests 
of ^each of these parties into account in evaluation design. Two pro- 
blems f aceo the researcher::, "knowing wt^at ultimately matters to each 
party.. .(and), knowing what findings from what research operations would 
be credible to, and usable by each party" (p. 294), The early identi- 
fication of these multiple publics is recommended, and establishment of 
a forum for continued interaction and dissemination of research results 
is suggested, although again einpirical evidence of the application of 
thi s approac h T s 1 ac ki ng . 

Froift an "open-system" perspective of institutions, evaluation is 
seen as a rational determination of fitness- for future survival and 
action in the facp of environmental uncertainty (Knapp, 1979b.). The 

11 • ■ 



institution's (or program-sO so.jircirf6r evidence of efjective ;func- 
tioning is dependent on what are perceived^to be salient and^v.isib^e 
criteria, by 'key actors within both the institution and its relevant 
environment* (Dill, 1962; Thompson, 1967). 'The selection of appropT4^^®' 
constitu€!ncies is limited to the identi,fication of central decision 
makers within- the institution and its immediate political context whose ^ 
^dndai:jls i^and beliefs about program impact are most influential* 

' "sUiehoTder analysis" (Ansoff , 1965; Ackoff, 1975) derives drigi- 
nally from ah objective- siting theory of the firm which "maintains that 



the objectives of the firm should be derived from balancing the con- . 

flirting claims of th^ various "stakeholders" ia-the firm: managers, ' 

workers, ^ckholciers, suppliers, vendors?^ The firm has a responsir,^'. 

' . . « . 

bility to. all "of these ,and must configure its objectives so as to give 
each a measure of satisfaction" (Ackoff, p,. 62). In evaluation research, 
the technique can 'her,- adapted to uncover key assumptions on program and 
evaluation goals from^ 'relevant audiences* for the evaluation, 'i-e, , tho^e 
managers, poliry makers, or afjfected parties perceived^ to be prinlal'y , 
users of l^luative' information, or to have some "stake" in the'evalua'- 
Tfoh-out^omesj^Mi^^ al. 1979). . ' ! ^ 

Approaches for identifying specific/ agencies/personnel in both ^ 
public and private sectors for stakeholder interviews^.shoil^d therefore 
be similar to those'an organization will employ to define itc own "irel- , 
evant environment", for example dist^ngu-ishing other agencies with which 
regular coniinuni cation is eslablished/or which are" concerne<? with eitlier 
. t^e flow of resources into, or services being delive^d by the pArogrcjm, 
agency that is involved in the evaluation. Strategies to determine the 



stakeholder universe therefore may include identi fyifig major organiza- 
tional representation :at legislative hearings concerning the program to 
be evaluated, and systematically inquiring of program administrators and 
(vlher, decisionmakers, in the program itself as to who should be <:onsid- 
ered stakeholders for the evaluation. Stakeholders might, be expected to 
include s^uch agencies as. those: (1) funding or qonducting the e^/alua- 

^tion; (2) administering, managing or delivering pri>^am services; (3) 
delivering related services;' (4) controlling or monitt>ring the flov7 of* 
funds into the program agency; (5) advocating or opposing services 

^delivered; and (6). professional societies or other externarorganiza-^ 
tions of agency personnel. 

B. Accessing Stakeholder Information 

Once stakeholders are defined, the following information can be 
sought (1) how stakeholders define the program and its objectives; (2) 
whether there is consensus or disagreement over appropriate measures of 
program performance; (3) what are stakeholders* particular information 
needs relative to^the current evaluation; and (4) what if any dissatis- 
factions with prior evaluations should suggest current improvements 

^(Wholey, 1973, pp. 53-54). Available methods for soliciting and ana- 
lyzing information range from qualitative to highly quantitative, de- 
pending on the instrumentation for the information survey and the re- 
sources for the analysis.^ Structured interviews^ offer techniques such 
as paired coniparispns, forced choice, Q sorts, Likert scales, or judge- 

' mental ranking schemes that provide ratio datatbased upon the subjective 
judgements of a number of individuals (e.g^„, Edwards^ 1971).* Suggestions 

. .?For a discussion- of the Jfard Edwards technique and its use in 
prioritizing public sector agency interview data, see Qrewes. et al., 
1976, pp. 17ff.) • ' ; 



have been nuide not onlyrLo prioritize audiences (Standards, 1900) but / 

also to prioritize information needs within audiences (Warcjo, 1981). 
Although rigorous quantitative schemes may be more desirable, 

factors af testing their _use include the following. Prioritizing mean- 
ingfully the between-agency responses implies either assigning equal , 
weights ta each response or prioritizing agencies and weighting accord- 
ingly. Agencies sponsoring, -as well as those conducting evaluations may 
be understandably reluctant to make overt their weightings across con- 
stjtuencies in this manner. Furthermore, it may prove more difficult to 
(1) find and access spokesperso'ns for stakeholder agencies under these 
cof^ditions, and (2) eljcit the frankness and depth in responses encoun- 
tered in less demanding frameworks. False precis'fon may be associated 
with stakeholder information collectejd and quantitatively , analyzed for 
example, without sampling'^a sufficient number of stakeholders within 
each constituency to assure data are representative of the .constituency 
identified. In/addition, development and pretest of the requisite 
instrumentation, requiring 0MB clearance if ton or more persons are 
fprmally surveyej^ through the use of identical, questions, (Office of 
Management and Budget, 1976) -can inhibit structured approaches and 
quantitative analyses. 

C'. Incorporating Results into the ^Evaluation Design^ 

The stakeholder survey can affect choice of an evaluation design 
from available alternatives both indirectly and directly. Indirectly, 
the development of 'study objectives, researclt questions and associated 
hypotheses are instrumental in determining the type of evaluation de- 

sign. Stakeholder emphases on particular target groups for study, on 

(J 

the procedural feasibility of certain data collection approaches, or on 



certain aspects of program service dejivery may influence, the selection 
of design variables.^ Alternatively, stakeholders may display biases ,pn 
concerns that reflect directly on chqice of study design, such as dis-^ 
satisfaction with previously used study designs, the sensitivity of 
particular measurement ^approaches or the length of time between baseline 
and follow-up data collection points. In the study reported 'here, the 
major impact was indirect through identification or refinement of study 
objectives, although some technical concerns were expressed by methodo- 
logitally oriented stakeholders towards such issuelas selection of 
appropriate controls, and representativeness of study\samples. 
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I -iii; IMI'LtMLNIlNG nil SIAKLIlOLUhK SURVllY 

The stakeholder' information assoSsincnt described \^ this paper was 
conducted during the initial stages of a multi-year evaluation of a 
national supplemental food program. ' • ^ . . ^ 

The purpose of the stakeholder surveys was tp.make the research 
team more knowledgeable about the key dimensions of the programmatic and 
political context within which the evaluation was to be conducted. 
•Major objectives therefore were to: 

— ^ define the appi^riate- stakeholder agency set; 

identify agency representatives and solicit information regard' 

^ ing ' . - . ' 

perceptions of the program, i^ 
expectations of the evaluation, 
information needs from the evaluation, and 
decisions lihe evaluation data should inform; \ ^ 

collect and analyze all data during the predesign phase of the 
evaluation study so as to permit 

" identification of research questions j/'^n^ 

subsequent refinement o.f evaluation^; objectives, and 
input into development of. an appropriate-^research design. 

The method used to gather and analyze the data consisted ^of four 

steps: 

identifying and selecting stakeholders for interview; 

selecting the method of access (telephone interview, in-person 
interview, written response); 

conducting the interview using an interview guide, or letter; 
arid * ' 

organizing and presenting responses. 
The research approach was that of a semi-structured, open-ended inter- 
view, with content analyses performed on the resultant data to derive 
central issues and research questions of importance to stakeholder 



constituencies. Specific proceiUires used to complete the above steps 
are discussed brielly- below. " \ 

' ■ \ \ ■ \- \ - 

. - ft 

A. ' Identi laying and Selecting Stakeholders \ 

A preliminary list of stakeholders and stakeholder agencies was 

generated through: ' 

review of legislative hearings involving the program agency, 
to identify sources of testimony; 

preliminary tracing—by review of documents (e.g., organ- 
iMtional charts, position statements) and discussion with 
agency personnel at different levels— of the flow of resources 
and information through federal, state, and local levels of 
program administration; 

discussion with a previously convened project advisory panel 
concerning relevant audiences for the evaluation; and 

prel iminary^ review of previous research on supplemental food 
program evaluation to determine important research constit- 
\' uencies. • - 

.Major stakeholder groupings emerged as: the agency conducting the evalu- 
ation;* agencies with legislative or execut^ive oversight for the program 
national, regional, state and local program administrators; and related 
professional or advo^cacy groups and associations., . 

After review of the list by project staff and the agency sponsoring 
the evaluation, the set of agencies was finalized consisting of agency 
personnel at federal and regional levels, congressional staff in appro- 
priate committees, members of the congressionally mandated National 
Advisory Council for the program, and representatiVes of eleven other 
federal and non-federal public and private stakeholder agencies, in- 
cluding various key advocacy and professional groups. 



*A separate evaluation arm of the overall agency administering the 
program. \ 
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One further method of idcnti lying stakeholders—asking respondents 

- — - - \ - " ' ' \ ' \ 

during the interview to recommend other key actors who should be coh- 

tacted--extended the original list'qut of railge, beyond available re- 
sources, arid necessitated an arbitrary cutoff point dictated primarily 
by the deadline date for a draft report. The bias created by the se- 
lection of stakeholders predominantly from federal or national consti- 
tuent agencies was recognized, and offset by other activities in the 
predesign \phase (e.g., a survey oT all 1,517 local program agencies, 
local agency fi^ld visits, and use of state and local agency consul- 
'tants). - " . 

B. Selecting the Method of Access ^ 

Three methods were used for interviewing stakeholders: in-pe>son 
interview, telephone, and letter. A national professional meeting for 
State program directors that also involved federal and regional repre- 
sentatives and a number of the stakeholder groups provided an opera- 
tionally efficient way to solicit information quickly through in-person 
interviews with many stakeholders. Telephone interviews were conducted-" 
with most other respondents, excluding National Advisory Council members 
or respondents who indicated they would rather respond in writing. Be- 
cause of their unique position and fjerspectives oa the program. National 
Advisory Council members were each sent a formal letter ^inviting their 
guidance and assistance in this part of the evaluation. In order to 
minimize the additional program-oriented burden on the respondents' 
time they, were further given the options of telephoning or mailing in 
the fF -responses. A small number of similar letters were sent to the. few 
respondents to the telephone interview who indicated a preference for 
written responses. ^ 



■ ■ ... -\ 

Conduct iiui the Interview 

A|^l personal and telephone InUt^iews, and letters to NationaT 
Advisory Council -members ' ollowed a similar format* The interview was 
preceded by a brief preanilile that explained the evaluation and pe 
purpose of the stakeholders* interviews in the context of that evalua- 
tion^ National Advisory Council members and telephone interviewees 
stating a preference for written responses v/erg sent a brief abstract of 
the evaluation project accompanying the lietter. All identified stake^ 

holderis, therefore, obtained similar information.upon^ contact, 

\ ■ 

Interviewers were in all c^ses experienced, professional project 
staff with key project responsibilities (task leaders). To maximize the 



utility of information obtainedj'n a few clear cases where it w^s appro- 
priate, interv^iewer expertise was matched to specific respondents (e,g., 
the task leader of the medical component of the evaluatic^n research team 
interviewed representatives of medical professional groups). Interview 
emphasis, stressed in pre-interview briefing sessions, was on the two 
chief components of the intervTiew: (1) respondent perceptions of the/ 
program, its original intent, and current context major changes in 
program focus, accomplishments and shortcomings* of the program; and (2) 
purposes the program evaluation, should serve in its current <^ontext, " 
specific decisions theevaluation could inform, and discVete stakeholder 
information needs the evaluation was expected to meet. Finally, respon- 
dents were asked to suggest other key constituencies or actors to be 
contacted as stakeholders. " * 

Respondents were encouraged to be frank and i^iformative, and each 
was assured that specific information would not be fo/mally linked to 
particular respondents in tne ensuing report. 



19 



Interviews were expecLed lo Lake approxiinaLely 3U ininuLes, and 
respondents were advised of this anticipated duration. Although no data 
were collected on actual interview lengths, i,t is estimated that the 

& 

mean interviev^^ time' for personal and telephone interviews appfoximated 
45 minutes. 



-4 



iV. RLSULIS i\k) RhCOMMtNUAl iONS 



^ - ■ •" 

A. Respondonts ' 

The type of stakeholder/agency from which responses were forthcoming 
and the method of access are illustrative of the coverage of tl^e survey. 
Forty-three stakeholders from. eleven different constituency groups were 



invited to peirticipate in the: int erviews. Thirty- four stakeho lders 

(79%) responded or were interviewed in time for their contribution to be 
jreflected„_in. the^study- Eighty-eight percent of the intervfews were 
even.ly divided between in-person and telephone interviews (fifteen each 
respectively) and the remaining four respondetl in writing. The break- 
down of respondents by agency and by type of response is as follows: 



Type of Interview 



Regional Program Offices 

State Program Offices 

Professional Associations 

Advocacy Groups 

National Advisory Council 

Federal Executive 
AgiBncies 

'I ' 

Federal Legislative 
^genc^es 



in"Pe)psdn 
. 7 



Telephone 

1 

1 , 

5 ■ 



Written Response 



1 



1 ■ 
2 

1 



Toft 



15 



2 

15 



A nationa1\^conference of program directors permitted in-person 
interviews with state and regional program administrators which other- 
wise would have hwtTcon^ucted by telephone. 



' . Because the actual subst.intue ^o&ults of this study are less 
central to the purpose of th\s paper than the methodological issu-;S 
raised as a function of conducting the survey, only** relevant references 
to the' data are included ^here. In the main study, results were orga- 
niz ed acc ording to major subject headings in the interviews, and . 
amounted to a considerable volume of d^ta presented in both tabular and 
textOal delaiT elsewhere (Lawrence 1980). A brief summary of the or* 
ganization of findings is included below, followed by a discussion of 

ways in which these findings were useful in refining evaluation ^objec- 

' 1 ' 1 . ' 

tives and in consideration of alternative designs. ^ 

Results were reported by five information categories, corresponding 

\ ^■ 

to the format of the interviews (program purpose, program context, , 
evaluation purpose, evaluation context and evaluation information needs) 
Narrative summaries were prepared of responses in the first four catego- 
ries, consisting of agency and individual perspectives of both the* pro- 
gram and the forthcoming Valuation. Evaluation information needs were 

firsrly categorized by stakeholder constituency into five substantive 

' \ 

groups as 'they related to: - , 

client health and nutritional status variables; 
client food and nutrition-related or health-related behaviors; 
programmatic behaviors; 
\ — . other client luediatimj ji/ariablcs; and 

not-elsewhere-classified information needs. 
Secondly, information needs were re-classified into the three 
components of a functional, service system model of the program: 



service production ami delivery; 
. — service utilization; and 

program outcomes • L 1 

No attempt was made to sort testable from untestable research 
questions at this point, since an important objective of this activity 
was to present information needs as they were currently perceived by 
stakeholders, and whether or not stakeholder expectations were realistic 
from a research standpoint. 

The narrative summaries of individual and agency perceptions of the 
purposes and operational contexts of both the program" and the evaluation 
provided a rich source of information on the various and quite differing 
perspectives of key stakeholders. Critical differences in^rceptions 
of program purpose were documented, for example, across federal agencies 
responsible for administration or oversight of, or coordination with the 
program. Greater consensus was noted in both the collective perceptions 
of evaluation purpose and respondent's ^support for the evaluation. 

The two break^outs of specific information needs permitted some 
estimation of the preponderance of information needs by category and by 
agency as they related to types of variables and program model compo- 
nents. Further prioritization was achieved by focusing on information 
needs of a subset of those^jublic sector agencies with immediate au- 
thority to direct program policy and administrative decision making. It 
was possible, thus, to identify more restrictively key information needs 

of those agencies most directly involved in shaping the future of the 

, • \ 

program. 



: _ •The-'data""Were useful in the following ways. Firstly, prior to 
completing the survey,* the evaluation team had only inchoate impressions 
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of issues and concerns central to agencies in the WIC environment inter- 
ested in or affected by the operation of ihe. program. Examination^of 
the program regulations and conversstions-with a limited number of 
program people had until the survey/ been the main source of infor- 
mation. The results permitted considerable expansion in and differ- 
entiation among both _soMrces and types of information relevant to de- 
signing a large scale field evaluation: perceptions of the program* 
expectations, of the evaluation, information needs held by various 
holderSj and the decisions the evaluation delta should inform. As such, 
the data were perceijved by the research team as a valuable part, but 
only a part of the, set of initial data gathered during the objectivie- 
setting stages of the evaluation. In particular, the information was 
useful . in, refining the conceptual model of the program originally de- 
veloped, from the literature review. ' 

1 

A second outcome was the opportunity to provide information on the 
evaluation to' those interviewed. The informal nature of the survey 
permitted the research team to explain evaluation purposes as developed 
to that point, and the general framework within which it would be con- 
ducted. Given the varying familiarity with the evaluation across var- 
ious stakeholder groups, a uniform description of the evaluation was 
thus provided with concomitant opportunities to correct inappropriate 
expectation or perceptions. Furthermore, the process of stakeholder 
interviewing permitted the initial generation of a network.of contacts 
sed-'repGcTteaiy for information and communication purposes throughout 
the early part of the study. 

Thirdly, while research ij|sues and questions identified through the 
survey clearly exceeded the resources available to explore them all in 

, I* - 



the current evalimLioi^. selecLUin ot Lho specific study objectives for 
the evaluation as well as development and refinement of the research 
hypotheses to be tested were directly informed by the survey results; 
Although other informatibh sources were used in this process, such as 
the literature review, an important part of the justification for study 
objectives- and hypotheses was the ability to document specific stake-*' 
holder interest (Cook et al. , 1981). Through the shaping of the evalu* 
ation objectives and' hypotheses the survey results impacted on the 
design and analysis strategy for the^ evaluation,* primarily th>'"^ugh 
suggesting key mediating and outcome variables of interest, as well as 
not only specific aspects of program delivery and utilization of ilmpor- 
tance to stakeholders, but also preferred study designs. For example, 
the implicit concern of major stakefiolders to obtain credible comparison 
subjects frpm comparable populations in areas not served by the program 
strongly encouraged the recommendation by the study team of a nonequiva- 
lent controj group design. [ 

Finally, stakeholders themselves appeared interested in the find- 
ings, particularly in how other stakeholders viewed the program, Ms - 
purposes and expected outcomes. Survey results also constitute a source 
pf suggestions for further research beyond the resource capacity Qf the 
present evaluation. In short, the survey accomplished its purpose, 
providing important information to guide t^he course of the evaluation, 
and providing also an identified set of constituencies with whom infor- 
mat ion might be exchanged as the evaluation progressed. 



Conclusions . . ' 

The^stakeholcler survey was onfe of four pre-design activities in a 
large, multi^year National evaluation** The survey took approximately 

one month to complete at 'a cost of aboirtr$10,000» which was less than 

* \ * ' . 

one-half of one i)ercent of total project resources* We have pointed out 

the usefulness of the results injfelping to set the research objectives 
for the evaluation* 

In the present study the informal , nature of the survey was empha- 
sized, in that no attempt was ina(ie to set priorities for , the information 
needs identified. This was in line with' the purpose of the survey: to 
be exhaustive in the identification of research issues to guide the 
evaluation* The unstructured interview format increased the likelihood 
that the resppndents would be less guarded in their responses** Informal 
conversations were judged more appropriate than^ fixed-response inter- 
views, since several of the interviews had to be conducted by phone* 
.Still, there may be situations where the researcher jWi 11 want to 
quantify survey responses and, thereby » attach priorHies to the. Infor*- 
mation n^eds identified. There may be a need, for example, to distin- 
guish among the information needs, or research issues, Jn terms of 
relat'Tve importance; project resources ^may then be allocated to infor- 
mation needs or research .issues that rank high in the total set identi- 
fied* While several priority ranking methocj are available as already 
"indicated, overt priority setting across agencies may bepoTitrically 
sensitive* , In addition, research teams need flexibility to, on occa- 
sion, weight implicitly, or even ignore a constituency with a particular 

axe to grind (Sechrest, 1981)* 

1 — " " 



*The other three were: telephone survey of all local agencies; 
review of extand data bases^^^mprehensive literature review and synthe-* 

sis. , ^ '"^ 



Finally, it is hoped that' Uiis study sets the stakeholder Purvey in 
some theoretijCal and empirical context, and can serve heuristically to 
guide future pijedesign efforts. lo the author* s experience prior to 
this study, there was more recognition of the need for stakeholder 
involvement in evaluation planning than practical guidance as to how 
best to obtain this i nvo 1 vfemeh t s "6i 1 1 less empirical evidence of, 
its advantages (Smith, i980). It is our feeling, having conducted thib 
survey, that efforts to define, identify and coopt constituency concerns 
into the design of major field evaluations are increasingly warranted. 
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